On Accuracy Benchmarks , Metrics , and Test Methods 1 Preface

This draft report was prepared by NIST staff at the request of the Technical Guidelines Development Committee (TGDC) to serve as a point of discussion at the Dec. 4-5 meeting of the TGDC. Prepared in conjunction with members of a TGDC subcommittee, the report is a discussion draft and does not represent a consensus view or recommendation from either NIST or the TGDC. It reflects the conclusions of NIST research staff for purposes of discussion. The TGDC is an advisory group to the Election Assistance Commission, which produces voluntary voting system guidelines and was established by the Help America Vote Act. NIST serves as a technical advisor to the TGDC. The NIST research and the draft report's conclusions are based on interviews and discussions with election officials, voting system vendors, computer scientists, and other experts in the field, as well as a literature search and the technical expertise of its authors. It is intended to help in developing guidelines for the next generation of electronic voting machine to ensure that these systems are as reliable, accurate, and secure as possible. Issues of certification or decertification of voting systems currently in place are outside the scope of this document and of the TGDC's deliberations. This document identifies problems with the test method for accuracy specified in VVSG'05 and describes some possible solutions. These possible solutions are the result of a preliminary analysis and do not constitute a NIST position or consensus. Harmonization with ongoing work on the similar topic of reliability testing has not yet occurred. We are providing this preliminary material only to keep the committee appraised of our activities and to provide opportunity for early feedback. The informal concept of voting system accuracy is formalized using the ratio of the number of errors that occur to the volume of data processed, also known as error rate. By keeping track of the number of errors and the volume of data over the course of a test campaign, one can trivially calculate the observed cumulative error rate. However, the observed error rate is not necessarily a good indication of the true error rate. The true error rate describes the expected performance of the system in the field, but it cannot be observed in a test campaign of finite duration, using a finite-sized sample. The system submitted for testing is assumed to be a representative sample (see [6] Ch. 8), so the …